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positive  definite  matrices.   The  methods  treated  allow  for  a 
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method  of  approximate  relaxation  is  introduced  which  includes 
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iteration  processes.   Explicit  error  estimates  are  given  in  cer- 
tain cases  and  examples  of  special  matrices  are  treated.   This 
extends  the  theorems  of  Temple  and  Ostrowski  to  group  relaxation 
methods  and  some  of  these  are  seen  to  be  feasible  for  use  by 
automatic  digital  computers. 

Relaxation  processes  with  arbitrary  orderings  are  shown 
to  converge  for  positive  definite  symmetric  matrices  and  also 
for  a  special  class  of  matrices  with  non-negative  inverses. 
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NYO  -  8673 
RELAXATION  METHODS  FOR  LINEAR  EQUATIONS 

1.  Introduction 

Let  A  =  (ai-?)  i»J  =  1,2, ...,n  be  a  real  non-singular 
matrix  of  order  n  >  1  and  consider  the  problem  of  solving 
the   equation 

(1.1)  r(u)    =     Au   -  b     =     o 

for  the  column  vector  u,   given  b,   by  an  iterative  pro- 
cedure.  Such  a  procedure  was  introduced  by  Gauss  [  Ij.  ] ,  to 
solve  (1.1)  and  is  commonly  referred  to  as  a  relaxation 
method  [3],  [11].   Among  the  matrices  to  which  the  method 
is  applicable  are  those  obtained  from  the  discretization  of 
certain  boundary  value  problems.   Southwell  [9],   who  gave 
the  method  its  name,  and  his  collaborators  have  applied  re- 
laxation methods  extensively  and  successfully  to  many  such 
problems  0 

The  method,  as  described  by  Gauss,  assumes  an  arbitrary 
initial  guess  u^°'  is  given.   One  of  the  n  scalar  equations 
of  (1.1),  say  the  i   ,  is  selected  and  the  i   component  u° 
is  replaced  by  the  number  uj  '   such  that  the  i   equation 
is  satisfied.   The  equation  selected  is  usually  one  which  is 
"worst,"  measured  by  some  fixed  norm  of  comparison.   This 
procedure  is  then  repeated  indefinitely. 
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A  method  where  the  choice  of  i   is  made  according  to 
the  largest  of  the   |r[°*|9  where   r^0'  =  Au^0'  -  b   is 
called  the  residual  vector ,  was  shown  to  converge  to  a 
solutions,  by  Temple  [11]  ,  for  positive  definite  symmetric 
matriceso   This  method  was  extended  by  Ostrowski  [6]  to 
allow  for  incomplete  relaxation,  that  is  under-  or  over- 
relaxation,  and  a  choice  of  three  norms  at  each  step»   He 
also  gives  an  error  estimate  for  this  process. 

The  theorems  given  below  extend  these  methods  to  group 
(or  block)  relaxation^  that  is,,  where  a  subset  of  components 
of  u     are  modified  at  each  step  by  the  inversion  of  a 
"small"  matrix „  The  incomplete  relaxation  is  then  replaced 
by,  what  may  be  c ailed,  approximate  relaxation,, 

Group  relaxation  techniques  are  found  to  be  more 
feasible  to  code  for  an  automatic  computer  than  the  scalar 
processes  and  seem  to  be  competetive  with  cyclic  processes 
in  time  of  operationo 

A  proof  is  also  given  for  the  convergence  of  a  process 

where  the  groups  are  chosen,  at  each  step,  in  an  arbitrary 

manner,.   The  class  of  matrices  to  which  this  applies  includes 

the  discrete  Laplacian  and  is  related  to  a  class  treated  by 

Ostrowski  [7]0  A  free  steering  theorem  is  also  given  for 
positive  definite  symmetric  matrices „ 


2.   Notation  and  Definitions 

Let   Z   denote  the  set  of  integers   (l,2,e..,n)   and  denote 
by  S   the  class  of  non-empty  proper  subsets   g   of   Z   such 
that  if  g  =  (i,  ,ip, . .  ,,i  )   then   i  <  i,  <  i~  <  <  ...  <  iv<  n, 
For  each  pair   g,h  e  S,  h  =  (j,  , . „ . ,  1  )   we  denote  a  submatrix 
of  A  by 


gh 


a 


iiJ 


a 


1J1 


iiJ 


1^2 


a. 


xvJl 


a.  . 
ivJ2 


a.  . 


l-*- 


SJ|i 


The  matrix  A    is  called  a  principal  submatrix  of  A  for 

any  g  e  S. 

Let  E   denote  the  linear  n-dimensional  space  of  column 
n 

T 
vectors   u  =  (u1,u2, . . . ,uR)    then  for  a  given  g  e  S 

u   =  (u1  ,u.  , oo»,u.  )    will  denote  a  v-dimensional  subvector 

g  xl      x2  v 

of     u. 

Let     ii   =    (g-.  ,g2>  •  •  •  »gjj)      denote   a   set    of   distince      g.    e  S, 

1   <   i   <   N      such   that 

N 
U        g,       =     Z      then     u     will   be   called   a  covering 
k=l  K 

of      Z      (or   simply  a  c ov er  ing ) .      If     it      is    a   covering    of      Z 

then     tz      is   called   a   partition   of      Z      (or  simply  a   partition) 

if        Si^gj    =  °,      i  !*  U      1  <   i,J   <  N,      and 

LJ    gk    =    z« 

k=l        K 
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If  it  is  any  partition  then  (lol)  may  be  written  as 

N 
re  (u)  hXZ  A     u   -  b   -  0 
si      k=l  sisk  sk    si 

g.  e  7t  9  i  =  1j>2j,oooPN  a 

In  particular s    if  we  denote  the  complement  of   g  by 
gi  =  2_g   then  for  the  partition  it  =  (g9g'),   (Id)  becomes, 
for  any  g  e  S 


rg(u)  ^Aggug+iWV  ~bg  =  0 


(2.1) 


rg»(u)  ^Ag«gug  +Ag«g«V 


b_„  =  0 


Let  K  =  (ca«)s   a  =  ls2soe.9v§  (3  =  1,2,. ..,n$ 
1  <  jx,v  <  n  be  a  given  matrix  then  for  the   g  and  h 
given  above  we  define  the  inflated  matrix  of  K  to  be 

[K]gh  =  (cpm)  p,m  =  1,2,..., n  where 

(2.2) 

(c    if  peg  and  m  e  h  and  p  =  o.a,  m  -  j„ 

°pm   / 

[0  otherwise  o 

If  ii  =  1,   ice,,   K  is  a  column  vector,  then  write   [K]  . 

Let  K  be  a  square  matrix  of  the  same  order  as  Agg 
then  it  follows  readily  from  the  above  definitions  that  for 
any  r  e  E 


n 


Tr~i  T 


(2.3)  r'M   r  -  rg  K  rg 


for  any  g  e  S  . 

Let   )S(p)jr)=o  bQ  a  sequence  of  sets   g(p)  e  S.   To 

each  p  =  0,1,2,...   associate  the  principal  sub-matrix 

A  t^\ni     \»   which  we  assume  to  be  non-singular,  and  let 
gvPJgipJ 

BP  =  Aglp)g(p)' 

We  \>j111  often   also  write    \  x   1      for   the    infinite 

PJ 

sequence  |x  I  ^.,   with  p  understood  to  range  from   0 

to   00  . 

Let  <  C j   be  a  secuence  of  matrices  of  the  same  orders 

P 

as  the  '   B   respectively,  and  let 

(2«5)  KP=BP+CP- 

Per  any  column  vector  u'    e  E   we  define  an 
approximate  relaxation  process  with  ordering  g(p)  by 

u(p+d  =  U(P) .      (p»  g  =  g(p) 

(2.6) 

g1  g' 

x^here   r<p)  =  r^u^)  . 

We  denote  the  solution  of  (1.1)  by  u"   and  we  shall 
examine  the  question  of  the  convergence  cf  \  u  P  J  to  u". 

If  I  u  "  ]  converges  to   u''  for  every  initial  guess 


u     and  every  b   we  will  say  that  the  process  (2.6)  is 
completely  convergent. 
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The  process  of  applying  (2.6)  will  be  called  a 
relaxation  step  or  briefly  a  relaxation,.   The  particular 
relaxations  obtained  when 


(2.7)  C   ■  e  B  p  =  0,1*2,0.0  , 

where   e   is  a  given  sequence  of  scalars,  is  referred  to 

by  Ostrowski  [6]  as  incomplete  relaxation  i.eo  overrelaxation 

(0  <  e  )   or  underrelaxation   (e  <  0)o   (The  numbers 

q   =    =  1  +  e   are  sometimes  called  relaxation  factors ). 
^P    P        P 

For  the  general  process  defined  by  (2Q6)  we  obtain  from 
(2,2)  that  the  residuals  are  iterated  by  the  rule  J 

(2.8)  r£P+l)  =  r£p)  -  AhgKpr^p)  ,   h  =  g,g«  ,   g  =  g(p) 
For  the  case  of  incomplete  relaxation  we  get  from  (2.7)  that 

r(p*i)  =  .  ,  r<p> 

g  P  8 

and  for  "exact"  relaxation,  that  is   6=0,   r^p    -  0. 

If  C    is  given  by  (2.7)  and  tc  =  (g-^ggs ° • ° pgjj)   is 
a  given  partition  then  the  process  given  by  (2.6)  is  the 
well  known  (see  for  instance  [53  »  [6],  [llj.3.  [16]   for 
group  iterations  and  [153,  [173,  for  the  scalar  case)  cyclic 
linear  group  iteration  if  g(p)   is  defined  by 

g(p)  =  gp+1  0  <  p  <  N 

(2.9) 

g(p)  =  g(p-N)  N  <  p 


9  - 


(Most  treatments  of  this  assume   e   to  be  a  fixed  number. ) 

P 

The  case  to  be  treated  here  is  one  where   g(p)   is 
computed  by  a  comparison  of  components  of  the  residual  vector. 

Let     it   =    (g-,,.. « >gvr)     be   a  given  covering  and     ||*|l     a 
given  norm  for     E        then  for   any     r    e  E       define 

(2.10)  i  =  £(ir,r,||»||) 

as   the   set      g.    e  it      (with  smallest   subscript      i)      such  that 

(2.11)  ll[r-]-||  =  max     ||[r      J      ||     . 

l<k<N         gk  sk 

Let     W    |j     be   a  given  sequence   of   coverings    such  that 

(2.12)  it      =    (gjp),...9g^p))    ,  1  <  N<  N   . 

P 

Let    ^  11*11/     \\     be   a  given  sequence   of  norms   and       \"*j    \    9yC  | 


/  \\       uc  a.  given  sequeiiue  v±     nurma    a_uu    <  - ;   v  ,  \  <- 

sequences   of   positive   constants   such  that   for  any     r    e  E 
7p  Tp    Irl2  <   ||r||fp)  <    7p|r!2 


0   <%VV<    7)  P   =   0,1,2,... 


(2.13) 

o  <  r 

P     P  -    /p 

i  1 2    T  — 

where   |r|   =  r  r.   Let  the  ordering  g(p)   be  defined  by 

(2.11;)         g  =  g(p)  =  ^(itp9r(p)J|#||(p))0 

Under  these  conditions  we  say  that  the  relaxation 
process  (2.6)  is  residually  ordered,  or  briefly  a  residually 
ordered  process  (r.o„pa)e 
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3o   Main  Theorem 

Unless  specified  otherwise  A  will  henceforth  be 
assumed  to  be  symmetric ,  although  some  of  the  results  hold 
also  for  Hermitian  matrices ,   For  the  r,o,p,  defined  above 
we  prove  t 

Theorem  le   For  the  ordering  g  =  g(p)   defined  by  the 
above  r0o0p0  let  the  matrices  A  9    g  e  it  ,  p  =  0j,l,2s<>.e 
be  positive  definite o   Let  a  sequence  of  matrix  valued 
functions  C  (rpsA?g)  =  C  ,   and  non-negative  numbers   6  „ 

Mr  Jt*  XT 

P  =  09l,2jo.o   exist,  such  that  the   C   are  of  the  same 

orders  as  the  A    and  such  that  they  satisfy  the  conditions? 

_ _        gg     __ — ___ -i_ _ — j. — — 

(3.D       (Vgp))T  WsP>  2  6P  <^P)>T  VsP)  •   s  =  g(p> 

(3o2)  0  <   6     <  1  for     p  =  0,1,25,0,, 

oo 

(3.3)  YZ.   (1-5    )T„     diverges. 

p=0  p        p 

The   r0oapa    defined  by   (2,6)   and    (2,1J4.)   is   then  completely 
convergent   if   and  only  if     A     is  positive  definite. 

If    (3q3)   is   replaced  by  the   assumption  that   there   exist 
positive  numbers      6      and      T      such  that 

(3.1;)  0  <  6     <  6  <  1 

(3.5)  o  <  t  <  rp  <  i 

then  the  above  conclusion  is  still  valid  and  the  sequence 
I  u  }  converges  to  u   like  a  geometric  series ,  i,e»,  there 
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exist  positive  constants   9   and   a   such  that 

|U(P>  _  u*|2  <  g  aP  |u(o)  .  u*|2 

(3.6) 

0  <  a  <  1   . 

The  proof  of  this  theorem  depends  on  several  leirsnas: 

Lemma  1.   Let   L    be  a  sequence  of  matrices  of 
p   , a-_ — 

order  n,   then  for  an  arbitrary  ev    s  E   define  the 
'     _ *-  n  _ — 

(P) 
sequence    e  *  by_ 

(3.7)  e(p+1)  =  Lpe(p)  ,  p  =  0,1,2,.., 

T 
Let  V(e)  =  e  Ae   for   e  e  E   and  assume  that 

(3.8)  V(e(p+1))  <  V(e(p))  ,         p  =  0,1,2, 


•  •  o   - 


If  lira  e^P*  =  0  for  every  initial   e^°*   then  A  is 

p->oo 
positive  definite. 

Proof;   Assume  that  for  some   e0'  ^  0,      V(e^0')  <  0 

then  for  the  sequence  defined  by  (3*7) 

0  >  V(e(o))  >  V(e(1))  >  ...  >  V(e(p))  >  ... 

Since   e  p}  — >  0   and  since  A   is  non-singular  V(e  p')  — >  0 
which  is  impossible.   We  therefore  have  that  V(e)  >  0  for 
all  e  e  E  .   Since   A  is  symmetric  and  non-singular  V(e)  ^  0 
for  e  ^  0   and  the  lemma  is  proved. 

Lemma  2.   Let   A  be  positive  definite  and9  for  a  given 
e  °   e  En,   let  the  L   and  e^p'  be  defined  as  in  Lemma  1 

-  12  - 


by   (3o'?)q      Assume   that   there   exists   a   sequence     la    i     such 

(o) 


that  for  any     e 


(3o9)  0  <  ap  <  1 

(3.10)  V(e(p+1))  <  ap  V(e(p))    I  p  =  0,1,2, 

and   assume    that 

as 

(3.11)  ZZ   (l~a    )        diver 
p=0  P 

then  lira  elp*  =  0  for  any  e^°' 


If  instead  of  (3.11)  it  is  assumed  that 


(3012)  0  <  a  <  a  <  1 

then  e  P   ~>  0  like  a  geometric  series. 

Proof g   Since  A   is   positive  definite  $,  there  exist 
positive  numbers  \D   A    such  that 

VeTa  <  V(e)  <  A-  ©T  e   » 

Prom  this  and  (3.10)  it  follows  that 

Xle(p+1)!2  <  V(e(p+1))  <  a  V(e(p))  <  8 >(o)!2A 

where  we  set   8  -  |  |  a.  o  From  (3.9)  we  obtain  that 

a  <  exp  (a  -1),   so  that  from  (3°H)  we  get  that  8   — >  0 

and  the  convergence  is  proved* 

If  on  the  other  hand  (3»12)  is  true  then 


13 
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(3.13)  |e(P>|2   <ap£    |e(0)|2    , 

or  the   convergence   is   geometric    and   the   lemma   is   proved. 

Proof   of  Theorem   1{      Let      uv         e  E       be    arbitrary  and 
define  first   the  process 

(3.1h)  u(p+1)   =  u(p)    -   Qpr(P) 


whe 


re      0      =  Qp{u(o),u(1),...,u{p),A,b),      p   =  0,1,...      is 


41 

a  given  sequence   of  matrices   of   order     n.      Let      u       be   the 
solution  of    (1.1)   and  denote   the   p        error  by     e'P     =  u' -    -u   « 
Let      L     =  I   -   Q  A    ,      where      I      is   the   unit  matrix  of   order 
n,      and   assume    that 

(3.15)  Qp(u(o),...,u(p),A,b)   =  Qp(e(o),...,e(p),A,0)    , 

p  =  0,1,2,... 

for  any     u      '      and     b    a      It   then  follows  from   (3.15)    and 
from     up  -u     =L     ep        that    the   convergence   of     u'P 

to      u*     is   equivalent   to    the   convergence   of      e   p        to      0 
where      e  *'      is   given  by    (3.7)   and 

L     =   L*{e(o)   +  u*,    ...,    e(p)   +  u*,A,b),      that   is 

(3.16)  e(p+1)  =   (I   -   eyO   e(p)      , 

(3.17)  r(p+1)   =   (I   -  AQp)   r(p)      . 
If  we  new  set 

-   Ik  - 


(3.18)  Qp  =  £Kp3gg  g  "  S(P)  » 

where   K   is  given  by  (2o5)s  then  it  is  clear  from  (2»2) 

that  the  relaxation  processes  (2o6)  are  of  the  form  (3«lij-)o 

Since  K   depends  on  u^p'   and  b   only  through  r^P'   in 
P 

the  r,Oopo9  (3olf?)  is  satisfied*   It  is  therefore  sufficient 
to  show  that  the  conditions  of  Lemmas  1  and  2  are  satisfied 
for  the   L   given  above o 

IT 

To  show  the  necessity  part  of  Theorem  1  we  verify  (3»8)<> 
We  will  find  it  convenient  at  times  to  drop  the  superscript 
p  and  write   (  )'   for   ()(p+1)?  g  for  g(p)  . 

We  note  that 

(3.19)  V(e)  =  eT  Ae  =  rT  A=1  r 


and  for  e8  =  e  -  G  r  =  L  e 

V(e»)  =  eTAe  -  2eTAG  r  +  (0  r)TACLr  so  that,  for  a   and 
K   given  by  (3.18)  and  (2*5) 

,(..)  .  v(e)  =  -2rT[Kp]ggr  ♦  rT[Kp]gg  A[Kp]ggr 

=  "2rg  Kp  *8  +  **  tKp*ggKp]ggr 
*  WggV^p'  r8 

■  *S(05-w°5*«v  rg 

=  rT(CTA  C  -B  )  r 
S>     P  gg  P  P   g 

from  (2,3)  and  (20i^). 
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Since  the  B   are  positive  definite  we  get  from  (3.1)» 
(3.2)  that 

(3.20)  V(e')  -  V(e)  <  (6-1)  i*gBprg  <  0  . 

(This  of  course  is  valid  for  any  ordering  g  =  g(p)   such 
that  (3.1)  and  (3,2)  hold).   Thus  we  have  the  conditions  of 
Lemma  1  satisfied  and  A  is  positive  definite. 

Assume  now  that   A  is  positive  definite  and  denote,  for 

any  g  e  S,   /1(A)  =  Ag»  x(Agg>  =  *g  to  be  the  largest 

and  smallest  eigenvalue  of  A    respectively.   In  particular 

gg 

set  /  V(A)  =  ■  •  ,  X(A)  =  %•      It  is  then  easily  seen  that 

(3.21)  *l*g<Ag</ 

and  A„„  is  positive  definite. 
Eg 

For  np  given  by  (2.12)  write  it  =  (g1,g2,..«,gTJ  ) 
then  for  g  =  g(p) 

A   rTB  r  >  |r  \2   =  |[r  ]  |2  >  -i  ||[r  ]  ||2  ,  > 

ne,  g  p  g  -  g'    ,l  g  g1  -  v.„  " l  gJg"  (p)  - 


N 


>  n~-  if   II  Ir   ]g  ll(p)  >  £  Irl2  >  -jP  V(e)  . 
p  p  k=l    sk  gk  vp'    p        wp 

Thus  it  follows  from  (3,20)  that 

(3.22)  V(e(p+1))  <  a  V(e(P})    where 

(3.23)  a_  =  1  -  (1-5  )  T  ~ . 

P  P     P  N  /\- 
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Since  the  /]        and  N   are  finite  in  number  we  get 
s  oo      p 
from  (3«3)  that        (1-a^)   diverges  and  the  convergence 

p=l 
follows  from  Lemma  2. 


p=0  p 


If  on  the  other  hand        ET       and      6        satisfy    (3.U)  and 

P  P 

(3«5)   then  let 

(3.21+)  a  =  l-(l-6)    ri,     ^  =  max  max  A      . 

^  P  geTtp        S 

Since     0<a     <a<l     we  obtain  from   (3.13)  that 
-     p  - 

(3*6)  is  valid  with     ©   -       /x,      and  the   theorem  is   proved. 

We  will  refer  to   the     a       defined  by    (3.23)  as   the 

P 

reduction  factors  of  the  r.o.p. 
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ko      Remark  1 

The  following  are  examples  of  norms   ||*||(  \   that  may 
be  used  In  practices 

(1)  (U.D        II  r  ||  =  |r|,   r-  1£«  1 

(2)  For   a  given  covering     it   =    (g1,g2» » . « 9gN)      let 

\i    =   max  /  \  _   ,     <$"  =   min   X 

l<k<N    sk  l<k<N  sk 

L__  Sk   8kgk  gk 
then  >  -  L      t-  -   ,_6*t  1 


,  =  N x-     ,_6*t  1 


If     it      is   a  partition,    then      >7    ■  ~\   ,       r  s      ( -£*)    * 
(3)      (U.3)  iirjj     -     y-      \r.\    p^=n,r=J. 


j=l 


(U)      ik*k) 


t=± 


||r  ||     =       max      |r.|      ,  ?  =  19°      n 
l<i<n        J 


These  norms   may   of  course  be  mixed   and   used   in  any   order, 
Remark     2„ 


For   the   particular   case  where   the   coverings     it        are 
partitions   and   are   all  the   same     it     s  it  =    (g-j,g?#  <>  •  •j.gjj)   we 
may  without   loss   assume   that    the      g,       are   consecutivej      that 
is      St    —    'lj   ^»    oeo,    i=)9      gp   =    ( i,    +  1  s    i=.    +2j,    o  o  o  s    ip  J,  o  o  »  9 
gjj  =    (iN  ,+lj,  i„      +  2,  a„o9n)o     It   is    then  convenient   to  denote 
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A 
ord 


glgj 


.  ,  \       —   X.   and  the 
1    Si    x 


A,  .  ;  u   =  u,  .   A    = 
ering   g(p)  =  g1^pj   by  the  integer   i(p)s  1  <  i(p)  <  N« 
The  matrix  A  may  be  written  in  the  forms 


All    A12 


L1N 


AN1    AN2 


LNN 


The  iteration  (2»6)  for  a  given  ordering   i(p)   may- 
then  be  written  in  the  form 

U(P+D   =  U(P)  .  K   (p) 

J  J       P  J 

(4-5) 


J  =  1(P) 
j  7*  i(p) 

where     K     -  A?7   +  C        and    the   residuals    iterate   by   the  rule 
p  li  p 

(4.6)  r<P+1>   =     r<*>>   -  AklV<P»    ,  i  .  i  (p)  . 

Let    the   ordering     g   -  g(p)      of   the  r0o0po   be  replaced 


by 

(l*o7)  i      =     m(p) 

that    is,      i      is   the   smallest    integer  such  that     Hfr^]        ||  /    \ 
is   maximal,    then  this    is   a  r0o0p0   but  with  a  fixed   partition, 


,(p)n 


For  the  special  case  where  C    is  given  by   (2«7)p 
(the  incomplete  relaxation  case),  such  that   -1  <  e  <  lp 
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p 
p  =  0,1,2,...$   let   6   =  e   .   It  then  follows  immediately 

that  (3.1),  (3.2)  are  satisfied  and  if 

00        o 

^ —  /-.  -^ 


(1-ep^p 


p=0 
diverges,  all  the  conditions  of  Theorem  1  are  satisfied. 

Ostrowski  [6]  has  proved  the  convergence  of  this 
r.o.p©  for  Hermitian  matrices,  under  the  assumption  that 
N  =  n,   that  is   A. .   is  simply  the  scalar  a. .   and  where 
the  norms  are  restricted  to  be  any  mixture  of  the  three 
norms  (1)  and  (2),   t  =  1,2. 

The  scalar  process  obtained  with  the  further  restriction 
that   e  =  0  or  C   =  0  was  shown  to  converge  by  Temple  [11], 
under  the  assumption  that  norm  (1)  is  used  exclusively. 

Theorem  1  allows  any  group  to  be  relaxed  if  it  contains 
an  equation  with  maximal  residual  in  absolute  value.   This 
is  obtained  if  each  it   consists  of  all  the  elements   g  e  S. 

It  should  also  be  noted  from  (3.6)  that  the  error  bound 
is  dependent  on  the  size  of  the  number    '  /\.      This  is  what 
is  called  the  P-condition  number  of  the  matrix  A   (see 
J.  Todd  [12]  )  and  may  get  quite  large  for  poorly  conditioned 
matrices. 
Remark  3» 

Suppose  that  between  each  relaxation  step  of  the  r.o.p. 
of  Theorem  1  a  finite  number  of  iterations  of  any  form  are 
taken  which  do  not  increase   V(e).   These  may  for  instance 
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be  relaxations  of  the  form  (2„6),  where  the   C    satisfy 
(3.1)  and  (3<>2),  and  are  taken  in  any  order.   This  would 
amount  to  extending  Lemma  2  to  require  only  that   0  <  a  <  1 
and  that  only  for  an  infinite  subsequence  is  (3<>9)  valido 
This  is  easily  done  and  implies  that,  except  for  the  error 
estimate  (3»6),  the  conclusions  of  Theorem  1  remain  unchanged,, 
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5.   Application  to  Compound  Iterations 

It  will  now  be  shown  how  the  approximate  relaxation 
method  of  Theorem  1  may  be  applied  to  the  practical  problem 
of  carrying  out  the  individual  relaxation  steps.   When  given 
by  (2.6)  these  involve  the  inversion  of  the  matrix  A    or 
solving  a  set  of  linear  equations,,   Even  if  the  inverse  of 
this  matrix  is  known  explicitly  (which  often  is  not  the  case) 
it  may  be  more  practical  to  compute   u  ^     only  approximately. 
Even  in  the  scalar  processes  it  is  a  common  practice,  for 
instance,  to  compute   u  P     only  to  a  few  significant  digits. 

Since  the  matrix  K   may  be  viewed  as  an  approximation 

to   A~'  s      with  the  allowable  error  measured  by  6   a  certain 

gg  *  ( 

amount  of  error  is  allowed  in  the  computation  of  u  **    .   It 

may  in  fact  be  convenient,  as  is  done  in  practice,  to  compute 

u'P     itself  by  some  iterative  process.   These  subsidiary 

iterations  are  sometimes  called  inner  iterations»   It  is 

indicated  below  how  these  iterative  processes  may  be  compounded. 


We  consider  the  following  compound  iterative  method > 
Let   u(o)  be  arbitrary  and  let  itp,  CpS  6p,  ||*||(p),^p9  Tp9 
g(p;  be  defined  as  in  the  r0o»p.  to  satisfy  (3«1)  and  (3«2)<> 

Suppose  u  =  u  P  ,   r  =  r^'  have  been  computed^,  we 

will  assume  u'  =  u  ^     to  be  computed  as  follows?   Determine 

g  =  g(p)   as  given  by  (2.11+)  and  set   u  ,  =  u  ,  ,   g1  =  Z-g„ 

g     g 

Let  v   be  the  solution  of 

(S.1)  .(*>  H  Agg(v-Ug^prg)  ♦  rg  =  0 
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and  let  v  be  an  approximate  solution  of  (5>ol)  such  that 

v  -  v  =  w  =  w(ps,A5r  )  depends  at  most  on  p9A  and  r  » 

We  then  set  ua  =  v   and  the  process  is  continued,, 
g 

Theorem  2o   Let  A.__9g  =  g(p)  be  positive  definite 

______   _         gg 

and  for  the  compound  process  defined  above 9  assume  that 
there  exists  positive  numbers  f  s  p  =  09l929oo»  such 
that 


(5»2)  0<6<yt)<1  P  -  0pls29ooo 

Jr  XT 

Yl/2=5l/2 

(5.3)  |w|    <      P  P     ■    |rgl  g  =  g(p) 

g 

and   such  that 

oo 

(5.U)  EZ  (i~Y_)T_, 


p=0 

then  the  above  compound  process  is  completely  convergent 
if  and  only  if  A  is  positive  definite,, 

If  instead  of  (5.U)  the   fp  satisfy  (3q5)  and 

(5o5)  0<Yp_EY<:L  P  =  0Pls29ooo 

then  u  P   converges  to   u   like  a  geometric  series0 

Proof  J   Let  ©  =  P   ■  P.  .  ,  then  we  exhibit  a  set  of 
_  ^g 

matrices  C   =  C    (r  P  9A9g)   and  numbers   6   such  that 

if  Jr  r 

the  conditions  of  Theorem  1  are  satisfied o 

Let  D   be  a  matrix  such  that  D  r  =  w.   This  matrix 
p  P  g 

exists  since   r  £   0  for  g  =  g(p),   and  depends  at  most  on 

O 
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r,   A   and     g.      We  may  therefore  write     v   =  v      +  D  r        or 

Ug  =  Ug   "    ( WVV      S   =  1(P)°      Let      °P  =  CP"DP     and 

d  =  r  B   r   •      Prom    (3»U»    (5*3)   and  the   consequent   inequali- 
g  P  g 

ties      |w|2  <  G2      d,       |A„  C   r    |2         A    6d      we   obtain  that 
-  g  ggPg       -        gP 

(C   r    )TA      (C   r    )   =   (C  r    )TA„    (C   r    )»2wTA     C  r  +wTA     w 
P  g        gg     P  g  P  g        gg     P  g  gg  P  g  gg 

<   5pd  +  2|w|     UggCprg|    *   Ag|w|2 

<  (6p+2eAg./^  *    |e2)d  =  (  ye;  +  'g9)2d  . 

If  we  let   5   =  (  v/5  +  /   G)   then  6  =  v  .   and,  since 
P       P    g  P    P 

y   satisfies  (5«2)  and  (5«U)  the  conditions  of  Theorem  1 
P 

are  satisfied.   If  y        satisfies  (5»5)  then  clearly  (3»k) 
is  satisfied  and  the  theorem  is  proved. 

For  the  particular  case  where  v   is  obtained  by 
successive  relaxations  we  have 

Corollary  1,   Let  a  subsidiary  r.o.p.,,  which  satisfies 

the  conditions  of  Theorem  1  be  given  for  (5>»1)  such  that 

the  initial  guess  is   v^°*  =  u^p*  -  C  r(p*.   Let   q  =  q  >  1 
a _ — ,         g      p  g     _  m   Hp  _ 

be  the  smallest  integer  such  that 

(5.6)  G    =  f  *  a     <  A)2  •  (  /v"  -  TO2 

q  -1   t=0  z  /[g  p     p 


where   the      a.       are   the    reduction  factors   of   the    subsidiary 
o»p.    ax 

=  Jq) 


r.o.p.  and  the  y^   satisfy  (5>2)  and  (5»J4-).   If  we  set 


in  the  above  compound  process  then  it  is  completely 
convergent  if  and  only  if  A   is  positive  definite,, 

Proof i      We  note  that   v  -  v   =  w   is  the  error  after  q 
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inner  iterations  and  is  dependent  at  most  on  A,   g  and 

on  the  initial  error  w^°'  =  B  s(v^°')  =  B  r  „   To  show 

P  P  g 

that  (5.3)  is  satisfied  we  use  (3.19),  (3-22)  and 

1      P, 

to  obtain 


|r  12 


q-i  %- 


■  Vl  i   i   Ad  "  ^o  i   i 
Ag    g      Ag     g 

Thus  the  conditions  of  Theorem  2  are  satisfied  and  the 
corollary  is  proved,, 

We  note  that  the  right  hand  side  of  (5o6)  depends 
on  the  P-condltion  of  A    and  again  limits  the  liberties 
we  may  take  in  the  process. 

In  some  cases  an  explicit  approximation  for  the  inverse 
of  a  matrix  A  may  be  used9  for  instance  A"  =  B  ^   2I-A  =  K. 
The  error  is  C  =  -R^  where  R  =  I -A  and  K  =  B+C.   This 
approximation  may  be  used  under  certain  conditions  as  indicated 
by  the  following? 

Corollary  2»   Let  a  r.Oop.  be  defined  for  (l.l)  with  the 
following  properties t 

1.  There  exist  non-negative  numbers   £  ,   p  =  0,l,2,ooo 
such  that  for  g  =  g(p) 

(5.7)  (Xl^p^g^Ag^p  +  l^        P  =  0,1,2,... 

00  ). 

(5»8)  £Z   d-C)r        diverges    . 

p=0  p        P 
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2.  Let  G  =  -R2B  where  R  =  (I-A)  .  Then  the 
resulting  r.o.p.  is  completely  convergent  if  and  only  If 
A   is  positive  definite. 

Proof;   Since  the   C    clearly  depend  at  most  on  A 
and  g   it  remains  only  to  exhibit   6  ,   p  =  0,l,2,..o 
such  that  (3»1)  and  (3,2)    are  satisfied. 

Using  the  same  notation  as  in  Theorem  2   we  get 

(C  r  )TA   (C  r  )  =  (R^r  )TB  r  <  /\(R^)d  <  ^d 
PS   gg  Pg      Pg   p  g  -  /  \ '  p '  -  *p 

if  we  let   6  =  C4"  we  have  form  (5.7)  and  (5.8)  that 
p   ^p 

(3.1),  (3.2)  and  (3.3)  are  satisfied  and  the  corollary  is 
proved. 

Similar  results  hold  for  higher  degree  approximations 
of  this  type. 
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6o   Arbitrary  Orderings 

For  certain  classes  of  matrices  the  ordering  function 
g(p)  may  be  chosen  in  almost  any  manner  and  still  the 
resulting  relaxation  process  converges  completely.   Such 
matrices  have  been  treated  by  Ostrowski  [7]  and  he  shows 
them  to  yield  complete  convergence  for  almost  all  arbitrarily 
chosen  orderings0   This  he  refers  to  as  a  free  steering 
method  in  distinction  to  the  steering  obtained  in  a  r<,0ope 

We  will  prove  a  "free  steering"  theorem  for  a  class  of 
matrices  somewhat  similar  to  Ostrowski* s,  but  allowing 
approximate  relaxations,. 

We  will  now  drop  the  assumption  that  A  is  symmetric 
but  still  assume  that  A  is  non-singular,,   For  any  m  by  n 
matrix  A  =  (a..)   we  write  A  >  0   if  sl±,  >  0S      1  <  i  <  m, 
1  <  j  <  n  l      and  call  A  non-negative „ 

We  define  (with  Collatz  [2])  a  matrix  to  be  monotone 
if  for  every  u  e  E   such  that  Au  >  0  implies  that  u  >  0„ 
It  is  clear  that  A  is  monotone  if  and  only  if  A   >  0o 

Theorem  3°  Let  A  be  monotone  and  g  =  g(p)  any  given 
orderingo  Let  {c  \  be  a  given  sequence  of  matrices  such  that 
for  p  =  0jlj2.,..,   and   g  =  g(p) 

i)  C   is  of  the  same  order  as  A  „   and   ~A„  C  >  0  9 

p  — — —   gg  —    gg  p  - 

->1 

ii)  A   and  K  =  A   +  C    are  non-singular  and  K  >  0, 
gg  —  P    gg    P  n  P  - 

iii)  ~Ag8g  >  0  o 
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Assume  further  that  the  sequence  i  g(p) f  covers  Z 
infinitely  often,  that  is,  for  each  i,  1  <  i  <  n, 
i  e  g(p)   for  infinitely  many  p. 

Then  the  iterative  process  (2.6)  is  completely  convergent. 
The  proof  depends  on  the  following; 

Lemma  3»   Let   A  be  monotone  and  j  Q  r  a  given  sequence 
of  matrices  of  order  n   such  that 

(6.1)  Qp  >  0 

(6.2)  Tp  =  I  -  A^  >  0         p=  0,1,2, 


•  ©  •   o 


Then  the  [  u  p  j   defined  by  (3»lU)  converges  to  some  vector 
in  E   for  any  u     and  any  b. 

Proof?   We  may  as  before  assume  b  =  0.   Assume  first 
that   e     is  given  such  that 

(6.3)  r(o)  =  Ae(o)  >  0   . 

Since     A      is  monotone      e(o^  >  0      and  from    (3-17)   and    (6.2) 
it   follows   that      r^p*  >   0      and      e^  >   0     for     p  =  0,l,2,..o. 
Since      e(p+1)-e(p)   =  -Qpr(p)  <   0     we  have    that      0  <   e(p+1)  < 
<ep     <8..<e      '      and   therefore    that      e   p        converges    to 
some   vector      e  o 

For  the    general   case  where      e  is    arbitrary,    let 

Ae  =  v   -  w      such   that     v  >   0      and        w  >  0*      Let     Av'        =  v 

and     Aw(o'   =  w     then     v(o'  >   0      and     w^°'   >   0.      Define   the 
sequences     [v(p)/,    !w(p);     by     v(p+1)   =  L  v(p),     w(p+1)-Lw(p) 
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then  by  the  above  result  v  P  9   w  p   converge  to   v    B 

w    '  respectively,.   Since  e  p'  =  v  p'  -  w^p'  the  lemma 

is  proved,, 

Remark  k° 

The  same  result  of  Lemma  3  Is  obtained s   by  a  similar 

proof,  if  instead  of  A  monotone  it  is  assumed  that  A  >  0 

and  (6„2)  is  replaced  by  I  -  QA  >  0  „ 

Proof  of  Theorem  Z%     We  note  first  that   Q  =  [K  ] >  0 

■ — — _____ — -*p  p  gg  _ 

and   since      (Qp)gg8    -    <Vgig  =   (Vg»g»    =  °     "  follows 
readily  that 

(Vgg  =  xgg  -  AggKP  "  "Agg°P  S  ••  (Tp'gg>  =  ° 
lVg'g  "  ~Ag'gKP  S  °  •  (Tp'g'g'  =  h'e<  s  °  • 

Thus  the  conditions  of  Lemma  3  are  satisfied  and  e  p  — *  e    « 

To  show  that  e     =  0  we  note  first  that  at  most  a 

finite  number  of  g(p)5  p  =  0plj,2Sooo   are  distinct  and  some 

subset  of  these  must  occur  infinitely  oftenQ   We  denote  this 

n 
(finite)    subset  by     P     and  for   any     r    e   E        let     ||r||  -  5~"|r.  |o 

n  fir=T 

Prom   (3ol6)  we  get   that 

P=t  P=t       ^ 

Assume  first  that  r^°'  >  09     then  rlp'  >  0  for  p  =  opls2SOoo 
Since  e  P   converges^,  it  follows  from  (6<,i|)  that 

p  =  II2Z  v(p)|!  =  £-  IV(p)|  ■  -=  I'V^co)11 

p=0  *       p=^0   T       p=0   p  glp; 


converges.   On  the  other  hand,  writing   Kf  for  K   when 
g(p)  =  f, 

P  > 

Plg(p)   =  f 
and  for  each  f  e  P   there  exists  an  infinite  subsequence 


<Pk) 


oo  .   Since 


(pA00        such  that      ||Kfrf   k    ||  — - *   0     as      k   — 
Kf      is   non-singular     r^  k'  — s»   0     for   every     f   e  P. 

Since   each     i,      1  <  i  <  n,      occurs    infinitely   often 

(Pk) 

there   exists   an     f   e  P     such  that      i   e  f      or   that     r^  q 

as      k  *oo.      Since     r(p)=Ae(p) >  Ae  (cD  )  =    r  (°°  ] ,      it 

follows  that  rjp) >  rj°°*  =  0  as   p  — »>  oo  or  e(co)=  0. 

If  r^0'   is  not  non-negative  we  define  v^p  ,  w^p'  as 
in  Lemma  3»   These,  by  the  preceding  argument,  converge  to 
zero  and  the  theorem  is  proved. 

Remark  5. 

It  follows  from  the  proof  that  the  condition  of  non- 
singularity  on  K-  could  be  replaced  by  the  condition  that 
for  each  f  e  F,  Kf  has  no  zero  column. 

It  has  been  shown  by  de  Rham  [8]  that  if  a  real  non- 
singular  matrix  A  with  non-positive  off-diagonal  elements 
(i.e.,  aj,  <  0  ij^j  1  <  i,j  <  n  )   has  no  real  negative 
eigenvalues  then  A  is  monotone.  This  is  an  extension  of 
a  theorem  of  Stieltjes  [10]  who  assumed  that  A  is  symmetric 
and  positive  definite. 
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Other  sufficient  conditions  for  monotonicity  have  been 
given  by  Collatz  [2]  and  Ostrowski  [73° 

Birkhoff  and  Varga  [l]  have  shown  that  if  A   is 

•=1 

symmetric  and   a„  ,  <  0  for  i  f   j   then  A    exists  and 

is  non-negative  if  and  only  if  A   is  positive  definite „ 

We  therefore  have  the  following 

Corollary  3«   Let  A  be  a  symmetric  matrix  such  that 
aii  >  °  *   ai-fi°»  $-  fi   U      1  <  i9j  <  n  then  the  following 
three  statements  are  equivalent? 

(lo)  Let   g(p)  be  any  ordering  that  covers  Z 
infinitely  often y  let   C   be  given  by  (2„7)  where 

completely  convergento 

(2o)  A   is  positive  definite 0 

(3o)  A  is  monotoneo 

Proof %      First  we  note  that  (lo)  implies  (2, )  by 
Lemma  ls  since  the  scalar  cyclic  processes ,  (that  iSg 
where   g(p)  =  i  ,   1  <  i  <  n  and  g(p)  is  given  by  (2»9) 
and  where   G  s  0) s      satisfies  the  conditions  of  that  lemmao 
By  de  Rham!s  (or  Stieltjes8)  theorem  (2o)  implies  (3o)o 

Since p  for  A  monotone 9    the  scalar  cyclic  processes 

will  be  completely  convergent  by  Theorem  2S  it  follows  from 

Lemma  1  that  (3»)  implies  (2»)o 

If  A  is  positive  definite  so  are  the  A  P   g  e  S9  and 

SS 

by  de  Rham^s    (or  Stieltjes8 )   theorem  the     A  are    also 

SS 

monotone0   Thus  all  the  conditions  of  Theorem  2  are  satisfied 
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and  (2*6)  is  completely  convergent.   Thus  (2„  )  implies 
(lo)  and  the  corollary  is  proved. 


-  31  - 


We  now  prove  a  free  steering  theorem  for  positive 
definite  symmetric  matrices,. 

Let  A  be  a  symmetric  matrix  and  jg(p)\  be  any 

ordering  such  that  it  covers  Z     infinitely  often  and  such 

that  the  matrices  A    are  positive  definite.   Let  1  e\ 

gg  L  P  i 

be  a  sequence  of  real  numbers  such  that  if  oo  =  l+er)  ^en 

(6.5)  Y  <  U)p  <  2-Y  .  0  <  Y  <  1  -> 

Let  the  matrices  \  d\      be  given  by  (2„7)  then  with  these 
assumptions  we  prove. 

Theorem  h<>      If  the  ordering  g  =  g(p)   is  such  thatfl 

for  each  p?   r  P   is  not  zero  then  the  relaxation  process 

defined  by  (2o6)  and  (2.7)  is  completely  convergent  if  and 
only  if  A   is  positive  definite. 

Proof |   If  we  let   6  ■  e^  then  (3»D  and  (3.2)  follow 

x'      x' 

from  (2.7)o   It  then  follows  as  in  the  proof  of  Theorem  1 
that 

(6.6)  V(e(p+1))  -  V(e(p))  =  (e^-l)(r£p))TB  r<p)  =  A   V  <  0 

P      &     PS       P 

The  necessity  then  follows  from  Lemma  1. 

To  prove  the  sufficiency  we  note  that  the  sequence 
^e^p  j   is  bounded  and  that  if  there  is  only  one  limit  point, 
that  is,  if  elp'  — >  x   then  x  =  0.   This  follows  from  the 
fact  that  the   V(e  p  )   decrease  mono tonic ally  to  the  limit 

V00  =  V(x).   It  then  follows  from  (6C6)  that  r(p)  — *>  0. 

S 

An  argument  similar  to  that  given  in  the  proof  of  Theorem  2 
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then  shows  that  r  P   — >  0.   It  therefore  suffices  to 
show  that   e'P    has  only  one  limit  point. 

Let   x  be  a  non-zero  limit  point  of  this  sequence  and 
W   a  sufficiently  small  neighborhood  of   x  which  excludes 
at  least  one  other  limit  point.   It  is  then  clear  that  for 
every  subneighborhood  U  of  x   there  exists  a  consecutive 
pair  e(p)  and  e(p+1)   such  that  e(p)  e  U  and  e(p+l)/W, 

Let   y  =  Ax   and  let   p  be  the  smallest  of  the  numbers 
|y.  |   where  the  y.   are  the  non-zero  scalar  components  of  y. 
For  a  sufficiently  small   e,   1  >  e  >  0,   let  W  be  defined 
by 

(6.7)  le  -  x|  <  — ■ 

and  if  r  =  Ae   then 

(6.5)  |r-y|  <  ep  <  elyl  . 

Let 

(6.9)  5  =min  (^fg^  >   *)  <  f 

where 

(6.10)  a  -  (|r|2  +  p)l/2  -  |rl 

P  =  I   p2  y2  (2-r2) 
and  define  the  subneighborhood  U  of  x  by 

(6.11)  !e-x!  <  2  . 

A 
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Let      e^P*    e  U     and     e^P+1'  ji  W     then  we   show  that 

y     /  0     for     g  =  g(p)«      For,    if  not,    then  from    (6.11)   it 
g 

follows   that      |r^p*-y|    <   5      and      |r^p'|    <   68      Prom   (2,6) 
and   (6,9)   it  follows  that 


e(t)   =  e(p)   -  ta  r(p)    ,  0  <  t  <  1 


■    (p+l)   _      (p),    m  ,B      (p),    <    (2-Y)5  <  £P 

|e  e         I         cOpl*yg      |   <       %  <  2A 

which,  together  with  (6.11),  implies  that 

|e(p+iLx|  <  ££ 

A 

This  however  contradicts  the  above  hypothesis  that  e  p 
is  not  in  W.   Thus  we  have  that 

(6.12)  f<  iyg|  -  6  <  !rgp)|  . 
Let   0   be  defined  by  (3„l8)  and  let 

■■% 

then  we  show  that  there  exist  values  of  t   such  that 

(6.13)  V(x)  >  V(e(t))  >  V(e(p+1))  . 

This  will  contradict  the  fact  that  the  V(e  P')   decrease 
monotonically  to   V(x). 
We  note  first  that 

(6.11;)  AV  =  V(e(t))  -  V(e(p))  =  t  u)Jtu)  -2)(r'p))TB  r[p)=k  A  V 

P     P      8      P  &     P  P 

and  distinguish  two  cases ' 
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(1)  Y   <  Jp   <   1 

(2)  1   <  O    <  2  -  Y  . 

In  case  (1)  is  follows  readily  that 

t(2-t,:  ) 

0  <  k   =  —5 -£-  <   1 

p        2.    -  „ 

or  that  _'  V  <  6  V  <   0  and 
P 

(6.15)  V(e(p+1))  <  V(e(t))  <  V(e(p))  . 

In  case  (2)  if   0  <  t  <  y/(2-y)   then  again  k  <  1  and  (6.15) 
holds. 

We  show  now  that  in  both  cases  there  exist  values  of   t 
such  that  (6.13)  holds.  We  note  first  from  (6.9)  and  (6.11)  that 

(6.16)  V(e(p))-V(x)   =  V(e(p)-x)  +  2(e(p)-x)T  y 

<Me(p)-x|2  +  *l£l 

<  ^-   (a2  +  2o|r|)   =  f 
Now,  in  case  (1),  if  y  <   t  <  1   then  from  (6.12)  and  (6.14) 

(6a7)         -AV  >  -  pt  (2-V)  ^    >  -6- 

which  together  with  (6.16)  implies  that  V(x)  >V(e(t)). 

In  case  (2),  if  y  <  t  <  y/(2-y),   we  find,  since 
.  t  <  1,   that  x  t(2  -  i,)  t)  >  t(2  -  t)   and  therefore  that 
(6.13)  again  is  valid.   This  gives  the  contradiction  and  the 
theorem  is  proved. 


-  35  - 


7<>   Convergence  Rate 

Let  the  rate  of  convergence  R  be  defined 9    for  a 
relaxation  process  (2a6)  with  a  fixed  partition 

(7.D     R  =   inf    11m  y  log  (!e/wt'lr)  ( 
e(o)gEnlt^oo  fe      le(Nt'l  3 

where   t  =  0,l,2j,..o  •   (This  definition  is  similar  to 
that  given  in  [1.3]  )»   If  this  is  a  r0o»p«  of  Theorem  1  we 
then  have  from  (3»22)  and  (3.21U  that 

n       Nt       -       ,   Nt 

(7.2)  R  >  llm  (-  i   log  TT  a  )  >  J  lim  (±  EZ  (1-6  )r). 

t->oo         p=0  p    ^  t-»a>   p=0     p  p 

Since   0  <  6  <1?   0  <  T  <  1  the  right  hand  side  will 

exist  o   If  T     =   1  and  6  =  6  <  1  then 
^P  P 

(7.3)  R  >  ^  N(l-6)  c 

r 

It  would  appear  from  this  that   6   should  be  chosen  to 
be  zero  but  this  is  true  (as  Ostrowski  points  out  in  [6]) 
only  tactically  but  not  for  the  over-all  strategy*, 

If  A  is  symmetric  and  positive  definite  and  g(p) 
is  a  given  ordering.,  it  follows  from  the  proof  of  (3.20)  that 

V(e<°>)-V(e<k>>=£(r<P>)WP)^v 


where 


p^O   S     P  • 


MP  =  BP  -  °PAggGP  '      S  =  g(p) 
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If  the     C        satisfy   (3-D   and    (3.2)    ohen      &v  >  0,      k  >  0. 

P  K  — 

The  problem  of  optimizing  the  process  for  a  given  r 

and  fixed  k   is  therefore  equivalent  to  making  a  choice 

for  g(p),   C    such  that  _'  ,  =  maximum.   This  "assignment" 
P  ^ 

type  of  problem  seems  quite  formidable  and  will  not  be 
treated  here. 

It  should  be  noted  that  the  lower  bound  given  by  (7.3) 
does  not  give  any  better  result  then  is  obtained,  say,  in 
the  cyclic  process  for  the  discrete  Laplacian.   This  should 
be  expected,  however,  since  a  r.o.p.  may,  at  times,  be  also 
simultaneously  cyclically  ordered,  as  will  be  seen  by  examples 
given  below. 
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8.  Examples 

We  consider  the  special  case  of  matrices  A  of  order 
vN  of  the  form 


VN 


J 

■I 

0 


-I 
J 

-I 


0 

-I 


0 


-I 


0 
0 


-I 


0 

-I 

J 


where   J  is  a  square  matrix  of  order  v,   and   I   is  the 

unit  matrix  of  order   v,   that  is,  we  have  the  fixed  partition 

case  %  =    (g-j,g2»  •  •  •  »Sm)   as  described  in  Remark  2#   This  form 

is  attained,  for  example,  in  the  discretization  of  a  Dirichlet 

problem  over  a  rectangle.   The  "groups"  determined  by  g,   will 

correspond  to  the  points  on  the  k    horizontal  line. 

In  the  examples  to  follow  we  assume  that  C   =0   and  the 

P 

norm  to  be  used  will  be  the  inner  product  (lj.,1).  For  a  given 
approximation  u,  N  numbers  |r. |  are  then  associated  with 
the  N  groups.  If  |r  |  is  maximal,  then  the  next  approxi- 
mation is  obtained  from* 


(8.1) 


u» 
m 


=  u   -  Hr 


m 


m 


"k  =  uk 


k/ 


m 
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where   H  =  J~  0   If  two  groups  have  the  same  associated  norm 
then  choose  that  one  with  smallest  subscript,,   (It  may  also 
be  useful  to  choose  the  one  nearest  the "boundary,"   that  is 
nearest   1   or  N  ) . 


(8.2) 


The   new  residuals    are   then  given  by  % 
i 


k 
t 
'k     =      0 

I 


rk  "■    ^m   » 


k-m  +  1,     l<m<N 


k  ~  m 

k  r  m,     m  +  1 


If  m  -   1   or  N  the  first  relation  of  (8.2)  is  valid  for 
k  =  2   or  N  -  1  respectively,. 

We  should  note  here  that  the  vector  Hp   determines 

m 

both     u»      and     r»      from    (8.1)    and    (8.2). 

We  will  now   consider   the   cases    of     N  =  2^3,      and      5 
to   estimate     V(e   ^    )      and   show   that   the     r.o0po      can   in  each 
of  these   cases   by  cyclic „      Before   proceeding  we  note   several 
properties    of    the  matrix     A„. 

Let     Fk(€)      be  the   matrix   of   order     k     of  the   form 


(8.3) 


Pk(0 


s 


>1 

0 
0 


0 


-1 


0 

.1 
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and  let   £k(£)  =  det  Pk^^#  Assume  that  A„  is  non-singular 
then  £N(J)   is  also  non-singular.   This  follows  from  the  fact 
that  the  eigenvalues  of  £N(J)   are   £w(^ )»   1  <  i  5  v» 
where  X.      is  an  eigenvalue  of  J.   If  for  some  \ . ,   say 

\  ,   such  that   Jy  =  X-,y,   y  ^  0  we  would  have   ^(^^  =  ° 

T  / 

then  there  exists  a  vector  x  =  (£-,»•••  »€N)  r   0   such  that 

F^  =  0.   Let  w  =  {%>-J^  ,*.*9K^^      then  w  /  0   and  by  a 
simple  computation  ANw  =  0,   Since  AN  is  non-singular  this 
is  impossible. 

If  we  now  let  A"1  =  GN  =  (gJ1*')  ,   1  <  i,j  <  N  then  a 
verification,  using  the  identities 


(8.J+) 


-^+1(J)  +  J^U)  -  S^U)  =  0  , 


k  >  1 


(N) 
shows  that   (K  .    is  given  by 


(6.5) 


<V   "  ^i-l(J)   W1   V'^ 

r(N)   ..  a(N)   _  a(N; 

Uij        '  Uji       '  UN+l-i,N+l-j 


i  <   j  <  N+l-i 


where  we  set  £  (J)  =  I.  The  identities  ( 8.i+ )  follow 

readily  from  the  computation  of  the   det  Fk+1 (£)• 

If  AN  is  symmetric  and  positive  definite  then 

2-(J),   1  <  k  <  N  are  non-singular.   Let   ♦>(£)  = 

=  2L   (£)/£,  -  (g)   then  tyk(J)   are  non-singular  and  we  may 

write  ^(J)  =  J,   \+1(J)  =  J-^CJ)"1*  k  >  1, 

£J€)  =  TT  +t(4)  and 
K       t=0  x 
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(8.6) 


aiV  ''  ^(J)  (    TT     ^U))-1,    ifj<N+i-i. 


ij 


k=l 


t=N-j+l 


Assume   now  that      N  =  2      and   let   a  r.o.p.   be   defined 

by   (8.1)   and    (1;.7)»      We  may  clearly  assume   that    the    initial 

( o  )  (o)  TT 

residual  vector     r  has   the   form     rv     '   =    (x    ,0)        so 

that   the   successive   residuals   are,    from   (8.2),   given   schema- 
tically by* 


s  P 

1 

0 

1 

2 

o  o  e 

1 

X 

0 

H2x 

2 

0 

Hx 

0 

or   that      r  ?    *    =  H  P  x.      This   process   is   clearly  both  cyclic 
and   residually  ordered » 


Since      d\^   =   U-U)]"1   =    (J-H)"1      and    since 


is 


"11  i.T2%"*J  vw  Ai/  ~^   "~"w~      "2 

positive   definite,    the   smallest   eigenvalue   of      J     is     X(J)=6>1, 

It  follows  also   from   (8.6)   that 

(8.7)  |e(P}|    =    lA-V^I    =    iHP^d-H2)"^! 


and  that 
(8.8) 


e 


(P) 


A(a2) 


(o) 


p  >   0 


(c^-Do-P"1    "     (cj^DoP"1 
Similarly  we   obtain  for  the   case   of     N  =   3      that   the 
resulting  r.o.p.    is   also    cyclic    if   the    initial    residual  has 

.s 


the   form     r(o)   =    (0,xT,0)T.      Since      gS£   =   J(J2-2I)"1     is 
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positive  definite   o"  >  fz     and  the  successive  residuals 
are  given  by  the  table  J 


\P 

0 

1 

2 

3 

0   • 

1 

0 

Hx 

0 

0 

2 

X 

0 

H2x 

2H2x 

3 

0 

Hx 

Hx 

0 

where  the  order  of  relaxation  is  2,1,3*2,1,3* • • •   since 

|H2x|  <  —  |Hx|  <  |Hx|.   The  special  form  required  for  the 

/2 
initial  residual  is  no  restriction  since  it  can  easily  be 

attained  by  two  preliminary  relaxations  of  the  first  and 

third  group.   It  follows  also  from  the  table  and  (8„6)  that 

for  p=3t,   t  =  0,1,2,... 

|e(P>|  =  lA-V^I  =  |H(I-ZH2r14P,l  "  U^I^H2)-^2^! 


or  that 
(8,9) 


k(3t) 


e 


2tA(A3) 


e 


(o) 


t  >  0  • 


For  the  case  N  =  5  we  obtain  a  r.o.p.  which  is  also 
cyclic  if  we  assume  that  the  initial  residual  has  the  form 
r(°)  =  (o,xT,0,xT,0)T  and  if  furthermore  0  >  3«  In  this 
case  the  table  obtained  is  I 
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1X\ 

0 

1 

2 

3 

k 

5 

9    ©   O 

1 

0 

Hx 

Hx 

Hx 

0 

0 

2 

X 

0 

0 

2H2x 

3H2x 

3H2x 

3 

0 

Hx 

2Hx 

0 

0 

0 

k 

X 

X 

0 

2H2x 

2H2x 

3H2x 

5 

0 

0 

Hx 

Hx 

Hx 

0 

where  the  order  used  is  %      2,1|.,3,1,5,2  , . .  „   .   Other  possible 

off  from  thi 
(o)  _   (o) 


forms  for  r     may  be  read  off  from  this  table.   If  r 


r^    ,  TA    '  =  r,;  '   then  by- 


has  the  symmetry  property  r,      ±  ^        »    x ?  '  *\ 

three  preliminary  relaxations  of  groups   1,5,3   in  that  order, 
the  desired  form  for  r     is  obtained.   The  error  growth 
can  again  be  estimated  for   p  =  5ts  t  =  0,1,2, ,0.   from 


(8.10)  |e(p)l  =   lA-yP*!   <   ©(-^je^l 

for  some  positive  constant   0  depending  only  on  ■&       . 


For  these  three  special  cases  we  obtain  from  (7»1)  that 


(8.11) 


N 
N 


=  2 


R   >   log 


3,   R 


i     ° 
log   2" 


^2 

N  =  5,  R  >  log  t~  • 

For  the  particular  case  of  the  discrete  Laplacian,  that 

2     it 
is,  when  J  =  Fv(lj.)   it  is  known  that   o  =  2  +  Ij.  sin  -glv   +\) 

and  the  rates  for   N  =  2   and   N  =  3   are   R  >  log  \\     and 
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E  >  log  2  respectively.   The  rates  of  convergence  of  the 
single  step,  cyclic,  successive  relaxation  process  (also 
known  as  Liebmann's  method)  for  these  cases  are,  asympto- 
tically  in  v  »  TB"  an(*   o?  respectively.   Thus  we  see 
that  the  line  methods  for  these  cases  are  better  than  twice 
as  fast  as  the  Liebmann  scheme.   This  was  shown,  by  Keller  ^ 
to  be  true  for  the  more  general  cyclic  process  using   N 
lines,  by  another  method. 

It  is  generally  believed  that  the  scalar  residually- 
ordered  processes  are  not  as  suitable  for  automatic  digital 
computers  as  the  cyclic  processes.   The  group  r.o.p.,  on 
the  other  hand,  being  somewhat  of  a  compromise,  is  more 
amenable  to  such  use.   This  method  was  in  fact  used  in  a 
code,  written  for  the  NYU  UNIVAC  I  by  G.  Rabinowitz,  to 
solve  the  Dirichlet  problem  for  a  rectangle  using  horizon- 
tal lines  as  the  groups.   Running  time  estimates  indicate 
that  the  r.o.p.  converged  at  better  than  twice  the  speed 
of  the  cyclic  line  relaxations,  the  sacrifice  being  that 
both  the  vectors   u  and  r  must  be  stored  simultaneously. 
There  is  only  a  small  increase  in  computation  since,  as 
pointed  out  above,  both  u!   and  r1   are  obtainable  from 

the  one  vector  Hr  .   The  r.o.p.  may  also  be  more  useful 

m 

where  greater  accuracy  is  needed,  since  error  estimates  are 
available  for  any  finite  number  of  relaxations  made. 
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